Adaptively optimizing order entry system

ABSTRACT

A system continuously improves the sensitivity, specificity, precision, and accuracy of treatment ordering templates. A repository of information comprises multiple candidate order sets individually including multiple candidate items for order and associated corresponding related order parameters. An individual item for order is associated with multiple related order parameters. A data entry monitor monitors user selection of candidate items from a candidate order set and records candidate item usage data identifying items selected by a user for order from individual particular candidate order sets for multiple different candidate order sets. A data processor determines from the candidate item usage data at least one of, (a) data indicative of the number or proportion of candidate items of a particular candidate order set that were selected by a user during order entry and (b) data indicative of the number or proportion of candidate items of a particular candidate order set that were not selected by a user during order entry.

This is a non-provisional application of provisional application serialNo. 60/988,120 filed Nov. 15, 2007, by J. DeHaan.

FIELD OF THE INVENTION

This invention concerns a system for adaptively updating templatemedical candidate treatment order sets for use by a physician, inordering treatment to be administered to a patient using a computerizedorder entry (CPOE) system, for example.

BACKGROUND OF THE INVENTION

A Clinical Information system (CIS) may use a large number of templates(such as clinical assessment forms and order sets i.e. lists ofcandidate treatment items for order for administration to a patient)that users can select to facilitate data entry and ordering treatmentfor a patient. A more advanced CIS may select the templates for the userand may also combine multiple templates into a single ad-hoc templatedeemed to be appropriate for a current situation. Templates used tocreate ad-hoc templates typically contain fewer items than templatesselected by the user. As a result, the number of templates used by amore advanced CIS is significantly higher (many thousands rather thanhundreds). This makes it difficult to identify which templates need tobe modified and what content or template selection criteria needs to bechanged to yield the highest benefit for overall use of the CIS.Anecdotal evidence and personal experience is no longer sufficient.

Known CIS systems do not compile statistics on how often template itemsare used, how often those items were not used and how many items wereadded by the users on an ad-hoc basis to complete a data entry task.System administrators and analysts use anecdotal evidence (like usercomplaints and change requests), personal experience or sometimesstatistics collected on an ad-hoc basis to identify which templatesshould be changed and what changes should be made to yield a highestcost-benefit trade-off.

Changes made to templates in known systems are typically sub-optimalbecause incomplete and sometimes misleading information is used todetermine what to change. Priorities are often based on subjectivecriteria (such as who complains the loudest) rather than on objectivecriteria that point to the highest benefits for an entire CIS usercommunity. Also, users are apt to accept without formally complaining,larger then necessary templates, because the required data is on thetemplate. It just takes more time the scroll up or down to find it. Thislengthens data entry tasks unnecessarily because opportunities to removeunused items from templates go unnoticed.

A system according to invention principles addresses the identifieddeficiencies and related problems and allows templates to be compared ina consistent manner and systematically identifies what templates need tobe changed and which changes yield the highest benefits.

SUMMARY OF THE INVENTION

A system continuously improves treatment ordering templates by makingmanual and automatic changes and deletions to the templates or the dataused for the automated selection or construction of such templates basedon usage statistics. A system adaptively updates template candidateorder sets. A repository of information comprises multiple candidateorder sets individually including multiple candidate items for order andassociated corresponding related order parameters. An individual itemfor order is associated with multiple related order parameters. A dataentry monitor monitors user selection of candidate items from acandidate order set and records candidate item usage data identifyingitems selected by a user for order from individual particular candidateorder sets for multiple different candidate order sets. A data processordetermines from the candidate item usage data at least one of, (a) dataindicative of the number or proportion of candidate items of aparticular candidate order set that were selected by a user during orderentry and (b) data indicative of the number or proportion of candidateitems of a particular candidate order set that were not selected by auser during order entry.

BRIEF DESCRIPTION OF THE DRAWING

FIG. 1 shows a system for adaptively updating template candidate ordersets, according to invention principles.

FIG. 2 illustrates interpretation of Receiver Operating Characteristicparameters.

FIG. 3 shows a scatter diagram presenting generalized template order setor order item, sensitivity (or True Positive Rate—TPR) plotted againstFalse Positive Rate (FPR or (1-specificity)) in a scatter diagram.

FIG. 4 illustrates a scatter diagram of treatment order template usagefor determining when and how to update a template, according toinvention principles.

FIG. 5 illustrates a scatter diagram resulting from shift in treatmentorder template markers resulting from weighting sensitivity relative tonew-specificity, according to invention principles.

FIG. 6 illustrates a scatter diagram indicating frequency of use oftreatment order templates by size of treatment order templaterepresentative markers, according to invention principles.

FIG. 7 shows a flowchart of a process used by a system for adaptivelyupdating template candidate clinical documents including order sets,according to invention principles.

DETAILED DESCRIPTION OF THE INVENTION

A system continuously improves the sensitivity, specificity, precision,and accuracy of treatment order templates used by clinicians who enterorders or clinical documentation into a computerized clinicalinformation system (CIS). A desirable order entry template (or otherdocument template) is one that contains the items a user requires forthe task at hand and no more. In that case the user does not need tomodify items, add items on an ad-hoc basis, or read and scroll throughitems that are not needed. The system systematically improves andupdates ordering templates by making manual and automatic changes totemplates by adding, deleting, changing or replacing order items in atemplate order set, for example, or the data used for the automatedselection or construction of such templates. The system adaptivelyupdates ordering and document templates based on usage statistics. Inone embodiment the usage statistics are compiled to provide a weightedReceiver Operating Characteristic (ROC), for example. The ROC enablesthe system to quickly identify templates and portions of a template thatwould benefit from updating. The data underlying the statistics identifycandidate items for deleting from a template, modifications to detailsof an item, and the items to be added to a template. The CIS uses thestatistics and underlying data to determine which items and item detailsto include in dynamically generating a template based on the inputprovided by a user preparing clinical treatment orders and documentationfor a patient.

A template as used herein comprises a form or displayed arrangement ofcells or data fields for presenting data items. A processor as usedherein is a device for executing stored machine-readable instructionsfor performing tasks and may comprise any one or combination of,hardware and firmware. A processor may also comprise memory storingmachine-readable instructions executable for performing tasks. Aprocessor acts upon information by manipulating, analyzing, modifying,converting or transmitting information for use by an executableprocedure or an information device, and/or by routing the information toan output device. A processor may use or comprise the capabilities of acontroller or microprocessor, for example. A processor may beelectrically coupled with any other processor enabling interactionand/or communication there-between. A processor comprising executableinstructions may be electrically coupled by being within storedexecutable instruction enabling interaction and/or communication withexecutable instructions comprising another processor. A user interfaceprocessor or generator is a known element comprising electroniccircuitry or software or a combination of both for generating displayimages or portions thereof. A user interface comprises one or moredisplay images enabling user interaction with a processor or otherdevice.

An executable application comprises code or machine readableinstructions for conditioning the processor to implement predeterminedfunctions, such as those of an operating system, a context dataacquisition system or other information processing system, for example,in response to user command or input. An executable procedure is asegment of code or machine readable instruction, sub-routine, or otherdistinct section of code or portion of an executable application forperforming one or more particular processes. These processes may includereceiving input data and/or parameters, performing operations onreceived input data and/or performing functions in response to receivedinput parameters, and providing resulting output data and/or parameters.A user interface (UI), as used herein, comprises one or more displayimages, generated by a user interface processor and enabling userinteraction with a processor or other device and associated dataacquisition and processing functions.

The UI also includes an executable procedure or executable application.The executable procedure or executable application conditions the userinterface processor to generate signals representing the UI displayimages. These signals are supplied to a display device which displaysthe image for viewing by the o user. The executable procedure orexecutable application further receives signals from user input devices,such as a keyboard, mouse, light pen, touch screen or any other meansallowing a user to provide data to a processor. The processor, undercontrol of an executable procedure or executable application,manipulates the UI display images in response to signals received fromthe input devices. In this way, the user interacts with the displayimage using the input devices, enabling user interaction with theprocessor or other device. The functions and process steps herein may beperformed automatically or wholly or partially in response to usercommand. An activity (including a step) performed automatically isperformed in response to executable instruction or device operationwithout user direct initiation of the activity. An object or data objectcomprises a grouping of data, executable instructions or a combinationof both or an executable procedure.

Workflow comprises a sequence of tasks performed by a device or workeror both. A workflow processor, as used herein, processes data todetermine tasks to add to a task list, remove from a task list ormodifies tasks incorporated on, or for incorporation on, a task list. Atask list is a list of tasks for performance by a worker or device or acombination of both. A workflow processor may or may not employ aworkflow engine. A workflow engine, as used herein, is a processorexecuting in response to predetermined process definitions thatimplement processes responsive to events and event associated data. Theworkflow engine implements processes in sequence and/or concurrently,responsive to event associated data to determine tasks for performanceby a device and or worker and for updating task lists of a device and aworker to include determined tasks. A process definition is definable bya user and comprises a sequence of process steps including one or more,of start, wait, decision and task allocation steps for performance by adevice and or worker, for example. An event is an occurrence affectingoperation of a process implemented using a process definition. Theworkflow engine includes a process definition function that allows usersto define a process that is to be followed and includes an EventMonitor, which captures events occurring in a Healthcare InformationSystem. A processor in the workflow engine tracks which processes arerunning, for which patients, and what step needs to be executed next,according to a process definition and includes a procedure for notifyingclinicians of a task to be performed, through their worklists (tasklists) and a procedure for allocating and assigning tasks to specificusers or specific teams. A document or record comprises a compilation ofdata in electronic form and is the equivalent of a paper document andmay comprise a single, self-contained unit of information.

FIG. 1 shows system 10 for adaptively updating template candidate ordersets. System 10 includes client devices (workstations) 12 and 14,repository 17 and server 20. Repository 17 (comprising one or more localor remote databases) includes information comprising multiple templatecandidate order sets individually including multiple candidate items fororder and associated corresponding related order parameters andassociates an individual item for order with multiple related orderparameters. Repository 17 also includes electronic patient medicalrecords, data representing recommended guidelines for treating differentmedical conditions, individual treatment order templates, medicaldocumentation templates, treatment orders placed by physicians forpatients and patient treatment plans and documentation indicatingcompliance with recommended treatment guidelines, for example. Server 20includes data entry monitor 25 for monitoring user selection ofcandidate items from a candidate order set and recording candidate itemusage data identifying items selected by a user for order fromindividual particular candidate order sets for multiple differentcandidate order sets.

Data processor 15 determines from the candidate item usage data at leastone of, (a) data indicative of the number or proportion of candidateitems of a particular candidate order set that were selected by a userduring order entry and (b) data indicative of the number or proportionof candidate items of a particular candidate order set that were notselected by a user during order entry. Update processor 29 automaticallyupdates a particular candidate order set by at least one of, (a)removing a candidate item, (b) adding a candidate item and (c) modifyinga candidate item. Update processor 29 may also (or as an alternative)automatically generate a message for communication to a user. Themessage identifies candidate items that may be removed from a particularcandidate order set and candidate items that may be added to aparticular candidate order set. Update processor 29 may also (or as analternative) automatically generate a (paper) report for communicationto a user. Clinical information system (CIS) 35 is a HealthcareInformation System (HIS) and includes a computerized order entry systemsupporting user ordering of treatment to be administered to a patientvia one or more display images provided by user interface processor 26on workstation 12 or 14. Clinical information system 35 presents formsto users via workstation 12 or 14 enabling a user to enter documentationfor the patient indicating data identifying which tests were ordered,the medical condition of the patient and reasons for orderingmedications or not ordering medications, for example. Workflow taskprocessor 40 prompts a user (e.g., healthcare worker) with order relatedtasks and to complete documentation, indicating an action taken by thehealthcare worker in treating the patient, and documenting compliancewith the recommended guidelines.

The system 10 devices are interconnected and bidirectionally communicatevia network 21 such as a LAN (Local Area Network) or other type ofnetwork. A client device (workstation) 12 or 14 includes user interfaceprocessor 26 and memory unit 28 and may comprise a personal computer,notebook, PDA, phone or other portable or fixed computerized processingdevice. System 10 may be used by a healthcare provider that isresponsible for monitoring the health and/or welfare of people in itscare. Examples of healthcare providers include, without limitation, ahospital, a nursing home, an assisted living care arrangement, a homehealth care arrangement, a hospice arrangement, a critical carearrangement, a health care clinic, a physical therapy clinic, achiropractic clinic, and a dental office. Examples of the people beingserviced by the healthcare provider include, without limitation, apatient, a resident, and a client.

System 10, in one embodiment, employs a Receiver OperatingCharacteristic (ROC) in processing data entry templates, e.g., fortreatment order entry or clinical documentation entry. For an individualtemplate, system 10 determines how often individual items in individualtemplates are used or selected in response to a selection command, howoften individual items were not used, and which items were added to thetemplate on an ad-hoc basis by a user. From this and other information,system 10 calculates sensitivity and new-specificity (as defined later)of an individual template. This information is plotted as a singlemarker on a scatter diagram, for example, which uses sensitivity andnew-specificity (or 1-new-specificity) as x and y axes. System 10 plotsindividual template sensitivity and specificity as a marker on a scatterdiagram for multiple templates.

Receiver Operating Characteristic (ROC) curves are known and used inmultiple fields including medicine to determine appropriateness ofdiagnostic procedures and clinical guidelines, for example. However,known systems fail to automate update of templates and templateselection criteria. System 10 applies objective criteria to multipletemplates (including large numbers of templates) to identify candidatedata entry template changes that will result in the highest cost-benefittrade-off for a CIS user population and prioritizes templatemaintenance. Update processor 29 automatically updates (or in oneembodiment partially automatically updates), template selection criteriaor template content. Data entry monitor 25 continuously monitors effectof automated template changes to ensure that they are appropriate.

System Definitions.

A Prediction is an item in a template displayed by system 10.

An Actual Value is an item also included in orders or documentssubmitted by a user.

A True Positive is an item in a template displayed by system 10 and alsoincluded in orders or documents submitted by a user.

A False Positive is an item in a template displayed by system 10 that isnot included in orders or documents submitted by a user.

A False Negative is an item that is not in a template displayed bysystem 10, but that is included in orders or documents submitted by auser after the user searched for it in a system catalog or entered it asfree text.

A True Negative is an item that is not in a template displayed by system10 and that is not included in orders or documents submitted by a user.

FIG. 2 illustrates interpretation of Receiver Operating Characteristicparameters comprising Prediction items (an item in a template) andActual Value items (a template item actually selected in Treatmentorders or clinical documents submitted by a user) in quadrantscorresponding to True Positive, False Positive, False Negative and TrueNegative.

Table of Definitions. true positive (TP) equivalent to a hit truenegative (TN) equivalent to a correct rejection false positive (FP)equivalent to a false alarm, (a Type I error) false negative (FN)equivalent to a miss, (a Type II error) true positive rate (TPR)equivalent to hit rate, recall, sensitivity TPR = TP/P = TP/(TP + FN)false positive rate (FPR) equivalent to a false alarm rate FPR = FP/N =FP/(FP + TN) accuracy (ACC) ACC = (TP + TN)/(P + N) ConventionalSpecificity CONVSPC = TN/(FP + TN) = 1 − FPR (CONVSPC) positivepredictive value equivalent to precision (PPV) PPV = TP/(TP + FP)negative predictive value NPV = TN/(TN + FN) (NPV) false discovery rate(FDR) FDR = FP/(FP + TP)

FIG. 3 shows general sensitivity (or True Positive Rate—TPR) plottedagainst False Positive Rate (FPR or (1-conventional specificity)) in ascatter diagram. In similar fashion to FIG. 2, templates scoring in thetop left corner are better and in the bottom right corner are worst andmost in need of automatic update by system 10. In using an ROC curve todetermine appropriateness of a diagnostic test or procedure, a test isdeemed to be sensitive if in a large number of cases the patient has themedical condition indicated by the test and if in a relatively smallnumber of cases the patient has the medical condition for which he orshe is being tested, even though the test indicates that the patientdoes not have the condition (false negative). System 10 advantageouslydeems a template as being analogous to a diagnostic test for the purposeof updating templates and documents. A template is deemed to besensitive if in a large number of cases most of the services ordered fora patient or most of the items documented for a patient are obtainedfrom the template and users on average need to add few items to atemplate order (or document) on an ad-hoc basis due to their absencefrom a template. A diagnostic test is deemed to be specific if in mostcases the patient does not have the medical condition when the testindicates that the patient does not have the condition, and if in arelatively small number of cases the patient has the medical conditionwhen the test indicates that the patient does not have that condition.Similarly, system 10 determines a template is specific if it does notcontain items users do not use and the number of items in a templatethat are not used by users is small. However, the number of items notused by users and not in the template is more or less unlimited andtherefore the conventional specificity measure (CONVSPC in the Table ofDefinitions) is meaningless in this context. Consequently, system 10advantageously uses a different definition of specificity,New-specificity*(SPC*)SPC*=1−FDR=1−(FP/(FP+TP)New-specificity (SPC) is the complement of the False Discovery Rate(FDR) and measures the number of unused items on a template relative tothe total number of items on the template. A small value for SPCindicates a better template that is less in need of update.

A template is very sensitive if it contains items representing all orsubstantially all, of the services users want to order and all orsubstantially all, of the items users want to document, but such atemplate is not very specific according to the new-specificity measure.A user needs to scroll through large numbers of items in the template tofind items to select. Conversely, a template is very specific accordingto the new-specificity measure if it contains substantially only itemsthat a user uses and not more, but such a template is not very sensitivebecause users often need to search a system catalog to find items thatare not in a template or need to enter an o item in free text. A goodtemplate therefore is a balance between sensitivity and new-specificity.System 10 predicts items a user wants to use in a template in a currentcontext. In one embodiment, system 10 provides ROC curves that are amirror image of conventional ROC curves and divide a scatter diagraminto four quadrants as illustrated in FIG. 2, to facilitatedetermination of when and how to update a template falling into aparticular quadrant.

FIG. 4 illustrates a scatter diagram of treatment order template usagedivided into four quadrants for determining when and how to update atemplate. The scatter diagram is a plot of new-specificity on thehorizontal axis against sensitivity on the vertical axis. Data processor15 (FIG. 1) uses data indicating candidate item usage to determine atleast one of, (a) data indicative of the number or proportion ofcandidate items of a particular candidate order set that were selectedby a user during order entry and (b) data indicative of the number orproportion of candidate items of a particular candidate order set thatwere not selected by a user during order entry. Data processor 15employs the determined data to provide the FIG. 4 scatter diagram. Ifthe particular candidate order set (a template) is in quadrant 403,update processor 29 removes infrequently used candidate items from theparticular candidate order set. If the particular candidate order set isin quadrant 405, data processor 15 performs a more detailed analysis ofcandidate item usage that is used to initiate candidate order setmodification by update processor 29. If the particular candidate orderset is in quadrant 407, update processor 29 adds candidate items enteredby users on an ad-hoc basis to the particular candidate order set. Ifthe particular candidate order set is in quadrant 409, data processor 15initiates a further analysis of usage of the particular candidate orderset

Low template sensitivity indicates a need for a user to enter many dataitems into a template on an ad-hoc basis because these desired items arenot available in the template. Low new-specificity indicates a need fora user to engage in extensive scrolling because the template contains alot of unused items that are available for selection just in case theuser needs them. Looking up items in the system catalog generally takesmore time than scrolling through a large template. Therefore dataprocessor 15 allocates sensitivity a different weight thannew-specificity in determining whether, and how, to update availableitems in a template. There are multiple ways to assign weight factors,by adding a number to a value, multiplying the number by a weight factorand making a scale exponential, for example. The effect of these dataprocessor 15 manipulations is to shift the positions of markersrepresenting templates in a scatter diagram towards one or anotherquadrant. This advantageously aids a user in viewing a presented scatterdiagram or aids update processor 29 in automatically evaluating scatterdiagram data, to identify which templates need updating.

FIG. 5 illustrates a scatter diagram resulting from shift in treatmentorder template representative markers resulting from weightingsensitivity relative to new-specificity. Specifically, FIG. 5illustrates a scatter diagram showing shift in the markers of the FIG. 4scatter diagram resulting from weighting sensitivity relative tonew-specificity. Many different shifts are possible depending on theweighting method employed, however, such shifts move the positions ofmarkers representing templates to advantageously facilitateidentification of templates to be updated. As a result of the shifts,more template representative markers are moved into quadrants 405 and407. This indicates more templates in quadrant 405 are categorized asrequiring detailed analysis of candidate item usage prior to initiatingcandidate order set modification and more templates in quadrant 407 arecategorized as requiring addition of candidate items entered by users.

FIG. 6 illustrates a scatter diagram indicating frequency of use oftreatment order templates by size of treatment order templaterepresentative markers. Large markers indicate high usage, therefore areallocated a high priority for template modification by data processor15. Specifically, FIG. 6 illustrates a scatter diagram showing relativefrequency of use of the FIG. 5 treatment order templates by size oftreatment order template representative markers.

For each set of treatment orders or clinical documentation templateitems submitted by a user, data entry monitor 25 compares the content ofa submitted set to the templates used by a user to construct (author)the content. The system labels the content of the submitted set (or acopy thereof) with different indicators. A first indicator indicates, auser copied an item from a template and identifies a template thatcontained the copied item. A second indicator indicates a user changedpart of a copied item (like an order attribute such as the dose of adrug) and for an individual item identifies whether a user selected thenew item content from a list of choices offered by the template, or ifthe user provided the item content. A third indicator indicates a usersearched a system catalog for an item and selected the item from thesearch results. A fourth indicator indicates a user entered the contentof an item without referring to data available in the system.

In response to a request, data entry monitor 25 compiles statistics fromthe labeled data. The compiled statistics include, the number of itemsselected or copied from a template with or without modification of thoseitems, the number of template items displayed but not used, the numberof items users added on an ad-hoc basis because they were not in thetemplate. These statistics may be filtered based on one or more criteriasuch as (but not limited to), manually selected templates versustemplates selected by system 10 at the point of use, specialty of auser, template selection criteria used by system 10 at point of use ofthe template (such as patient age and gender, identified patientproblems, goals) and time period during which the templates were used.System 10 may exclude from analysis those template items that have beenidentified as mandatory due to regulations or hospital policies. In oneembodiment user interface 26 displays or renders ROC curve scatterdiagrams as previously described facilitating user selection oftemplates to be updated by update processor 29. In another embodiment,data processor 15 automatically analyzes scatter diagram data toidentify templates to update and the data items to be added, deleted ormodified and update processor 29 automatically updates identifiedtemplates and data items based on the analysis. Data processor 15 doesthis by identifying templates in particular quadrants via predeterminedquadrant thresholds, for example. Data processor 15 identifies items tobe automatically added, modified and removed based on statisticsrecorded by data entry monitor 25 and automatically adds, modifies orremoves items based on template quadrant.

Alternatively, data processor 15 applies (modifiable) rules todetermined sensitivity and new-specificity statistics and underlyingdata to determine which changes system 10 needs to make to whichtemplates to improve overall quality and usability of treatment orderingand clinical documentation. Data processor 15 produces ROC curve scatterdiagrams for use in monitoring the system ability to automaticallyimprove and update data entry templates. In addition, besidessensitivity and new-specificity values, data processor 15 producesstatistics like precision, accuracy and others previously described toproduce scatter diagrams that 1take two variables into account. In oneembodiment, system 10 compiles and filters template usage data andpresents the sensitivity and new-specificity statistics in ROC diagrams,for example. However, the same template usage data collected by system10 is also used to calculate other measures of template candidate orderitem quality. Examples of such other measures include, the total oraverage number of times treatment orders or documentation itemsbelonging to a particular template were presented to a user in aparticular time period, but that were not used for patient care and thetotal or average number of number of treatment orders or documentationitems users had to find in the system catalog or enter as free textwhile the template was presented to the user. Additional determinedmeasures comprise ratios of these other measures.

In another embodiment, system 10 collects template usage data andcomputes and stores summary statistics like total number of templateitems not used for each usage of a template and uses the summarystatistics without retaining the information about which specific itemswhere not used. Other embodiments may collect detailed template usagedata, but stores the collected data separately (e.g., in a log file)from the template or separately from the treatment items ordered ordocumented for a particular patient while maintaining references to thetemplate or the order or clinical documentation session.

FIG. 7 shows a flowchart of a process used by system 10 for adaptivelyupdating template candidate clinical documents including order sets.Further, in one embodiment a clinical document comprises a treatmentorder set. In step 702, following the start at step 701, data processor15 stores information in repository 17 comprising multiple candidateclinical documentation sets (e.g. template order sets) individuallyincluding multiple candidate documents (or orders) for clinicaldocumentation and associated corresponding related clinicaldocumentation parameters (e.g., treatment order related parameters). Anindividual document for clinical documentation being associated withmultiple related clinical documentation parameters. Data entry monitor25 in step 704 monitors user selection of candidate items from acandidate clinical document (e.g., order set) and in step 709 recordscandidate item usage data identifying items selected by a user for usein individual particular candidate clinical documents (e.g., candidateorder sets) for multiple different candidate clinical documents. Dataentry monitor 25 also records document usage data identifying documentsselected by a user from individual particular candidate clinicaldocumentation sets for multiple different candidate clinicaldocumentation sets. Data entry monitor 25 monitors orders and clinicaldocument items entered by a user in addition to selecting candidateitems from a candidate order set or candidate document template.

In step 714, data processor 15 determines from the recorded candidateitem usage data parameters comprising at least one of, (i) dataindicative of the number or proportion of candidate items of aparticular candidate clinical document (e.g. order) that were selectedby a user during data (e.g., order) entry and (ii) data indicative ofthe number or proportion of candidate items of a particular candidateclinical document (e.g., order set) that were not selected by a userduring data (e.g., order) entry. The data indicative of the number orproportion of candidate items of a particular candidate order set thatwere selected by a user during order entry comprises data indicative ofthe number or proportion of items ordered by the user that were in acandidate order set. Data processor 15 also determines from the recordedcandidate document usage data parameters comprising at least one of, (a)data indicative of the number or proportion of candidate documents of aparticular candidate clinical documentation set that were selected by auser during clinical documentation entry and (b) data indicative of thenumber or proportion of candidate documents of a particular candidateclinical documentation set that were not selected by a user duringclinical documentation entry. Data processor 15 further determines dataindicative of the number or proportion of items ordered by a user thatwere not in a candidate order set, identifies individual candidate itemsof the candidate item usage data that were most frequently not selectedand identifies individual candidate items of the candidate item usagedata that were most frequently selected.

In step 719, data processor 15 identifies a candidate clinical document(e.g., a candidate order set) to be updated in response to thedetermined parameters including a weighted Receiver OperatingCharacteristic (ROC) derived using the determined parameters. Dataprocessor 15 identifies multiple candidate clinical documents to beupdated in response to the determined parameters and prioritizescandidate clinical documents to be updated based on frequency of use ofuse of the candidate clinical documents. Data processor 15 furtheridentifies an individual candidate item of a particular candidateclinical document that was not selected by a user during data entrybased on a frequency of non-selection exceeding a predeterminedthreshold non-selection frequency or for a proportion of times exceedinga predetermined proportion. In one embodiment, user interface 26presents in a display image a diagram (e.g., a Receiver OperatingCharacteristic) indicating at least one of, (a) candidate items that maybe removed from a particular candidate order set and (b) candidate itemsthat may be added to a particular candidate order set. In step 723,update processor 29 automatically (or in one embodiment in response touser command) updates the identified candidate clinical document (e.g.,order set) by at least one of, (a) removing a candidate item, (b) addinga candidate item and (c) modifying a candidate item. Update processor 29also automatically generates a message or report for communication to auser identifying at least one of, (a) candidate items that may beremoved from a particular candidate order set and (b) candidate itemsthat may be added to a particular candidate order set. The process ofFIG. 7 terminates at step 731.

The systems and processes of FIGS. 1-7 are not exclusive. Other systems,processes and menus may be derived in accordance with the principles ofthe invention to accomplish the same objectives. Although this inventionhas been described with reference to particular embodiments, it is to beunderstood that the embodiments and variations shown and describedherein are for illustration purposes only. Modifications to the currentdesign may be implemented by those skilled in the art, without departingfrom the scope of the invention. Further, the processes and applicationsmay, in alternative embodiments, be located on one or more (e.g.,distributed) processing devices. Any of the functions and steps providedin FIGS. 1-7 may be implemented in hardware, software or a combinationof both.

What is claimed is:
 1. A method for adaptively updating templatecandidate clinical documents, comprising the activities of: employing atleast one computer for, monitoring user selection of candidate itemsfrom a candidate clinical document template; recording candidate itemusage data identifying items selected by a user for use in individualparticular candidate clinical document templates for a plurality ofdifferent candidate clinical document templates; determining parametersfrom compiled statistics derived from the recorded candidate item usagedata, said compiled statistics comprising at least one of, (i) dataindicative of the number or proportion of candidate items of aparticular candidate clinical: document template that were selected by auser during data entry and (ii) data indicative of the number orproportion of candidate items of a particular candidate clinicaldocument template that were not selected by a user during data entry;identifying a candidate clinical document template to be updated inresponse to the determined parameters; and adaptively updating theidentified candidate clinical document template by at least one of, (a)removing a candidate item, (b) adding a candidate item and (c) modifyinga candidate item.
 2. A method according to claim 1, including theactivities of compiling said statistics derived from the recordedcandidate item usage data and identifying a candidate clinical documenttemplate is to be updated in response to a weighted Receiver OperatingCharacteristic (ROC) derived using the determined parameters.
 3. Amethod according to claim 1, including the activity of identifying aplurality of candidate clinical document templates to be updated inresponse to the determined parameters and prioritizing candidateclinical document templates to be updated based on frequency of use ofuse of the candidate clinical document templates.
 4. A method accordingto claim 1, including the activity of identifying an individualcandidate item of a particular candidate clinical document template thatwas not selected by a user during data entry based on a frequency ofnon-selection exceeding a predetermined threshold non-selectionfrequency.
 5. A method according to claim 1, including the activity ofidentifying an individual candidate item of a particular candidateclinical document template that was not selected by a user during dataentry for a proportion of times exceeding a predetermined proportion. 6.A method according to claim 1, wherein said clinical, document templatecomprises a treatment order set.
 7. At least one computer system foradaptively updating template candidate clinical documentation sets,comprising: a repository of information comprising a plurality ofcandidate clinical documentation sets individually including a pluralityof candidate documents templates for clinical documentation andassociated corresponding related clinical documentation parameters, anindividual document template for clinical documentation being associatedwith a plurality of related clinical documentation parameters; a dataentry monitor for monitoring user selection of candidate documenttemplates from a candidate clinical documentation set acquired from therepository and recording candidate document template usage dataidentifying document templates selected by a user for clinicaldocumentation from individual particular candidate clinicaldocumentation sets for a plurality of different candidate clinicaldocumentation sets; and a data processor for, determining parametersfrom compiled statistics derived from the recorded candidate documenttemplate usage data, said compiled statistics comprising at least oneof, (a) data indicative of the number or proportion of candidatedocument templates of a particular candidate clinical documentation setthat were selected by a user during clinical documentation entry and (b)data indicative of the number or proportion of candidate documenttemplates of a particular candidate clinical documentation set that werenot selected by a user during clinical documentation entry, identifyinga candidate document template to be updated in response to thedetermined parameters; and adaptively updating the identified candidatedocument template by at least one of, (a) removing a candidate item, (b)adding a candidate item and (c) modifying a candidate item.
 8. A systemaccording to claim 7, wherein said data processor, compiles saidstatistics derived from the recorded candidate item usage data andidentifies a candidate clinical document template is to be updated inresponse to a weighted Receiver Operating Characteristic (ROC) derivedusing the determined parameters.
 9. A system according to claim 7,wherein said candidate document template comprises a treatment orderset.